{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "00_pytorch_fundamentals_video.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "authorship_tag": "ABX9TyMHZOb/mEFxyxNc+HBmyfpJ",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mrdbourke/pytorch-deep-learning/blob/main/video_notebooks/00_pytorch_fundamentals_video.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 00. PyTorch Fundamentals\n",
        "\n",
        "Resource notebook: https://www.learnpytorch.io/00_pytorch_fundamentals/\n",
        "\n",
        "If you have a question: https://github.com/mrdbourke/pytorch-deep-learning/discussions"
      ],
      "metadata": {
        "id": "3Wh_8cvFcDK9"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "37E0a1dNbfa4",
        "outputId": "e94ca35e-c228-4334-c1ac-92cd9939ac68"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1.10.0+cu111\n"
          ]
        }
      ],
      "source": [
        "import torch\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "print(torch.__version__)"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Introduction to Tensors\n",
        "\n",
        "### Creating tensors\n",
        "\n",
        "PyTorch tensors are created using `torch.Tensor()` = https://pytorch.org/docs/stable/tensors.html"
      ],
      "metadata": {
        "id": "JvYzOsXRdAVv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# scalar\n",
        "scalar = torch.tensor(7)\n",
        "scalar"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UTWI01UEdvni",
        "outputId": "8ef0f4a9-7cd4-40e2-e7b9-cfefeb9f221f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(7)"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "scalar.ndim"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5OQ5Mrxgd80W",
        "outputId": "dc58e0fd-04a7-4d9c-8b88-14df0436a8a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get tensor back as Python int\n",
        "scalar.item()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CUDzqK6FehIE",
        "outputId": "7c609b44-1e24-4629-e8bb-6711f814f27e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "7"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Vector\n",
        "vector = torch.tensor([7, 7])\n",
        "vector"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LV0-tARqelWv",
        "outputId": "6ad075f9-fad5-4d96-db28-e0b878334ffb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([7, 7])"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "vector.ndim"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "whsizerteuKT",
        "outputId": "6c09eaf9-730a-4dce-e36d-48935a02dad3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "vector.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FV6aI5Cpe6VT",
        "outputId": "271b0a0a-05b0-410c-de4c-1356f8640a31"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([2])"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# MATRIX\n",
        "MATRIX = torch.tensor([[7, 8],\n",
        "                       [9, 10]])\n",
        "MATRIX"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NOTLQBwye_wo",
        "outputId": "afdc9fb9-a8cd-4d21-de3f-5473d7fb952b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 7,  8],\n",
              "        [ 9, 10]])"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "MATRIX.ndim"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "3ArSidOCfPc_",
        "outputId": "350b3e30-f07f-4207-c49a-a09e099101a5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "2"
            ]
          },
          "metadata": {},
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "MATRIX[1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BiIdkyk6fWhc",
        "outputId": "2d7d2c70-13b4-4da9-9720-e20db452389c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 9, 10])"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "MATRIX.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QW1d-fKDfZHu",
        "outputId": "c0a7b2ba-6d21-41c3-b85f-85fc57ef8fb9"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([2, 2])"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# TENSOR\n",
        "TENSOR = torch.tensor([[[1, 2, 3],\n",
        "                        [3, 6, 9],\n",
        "                        [2, 4, 5]]])\n",
        "\n",
        "TENSOR"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GxOyqq8XffEx",
        "outputId": "a2e4ae17-c902-4656-d8fd-63deb9a219e6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[[1, 2, 3],\n",
              "         [3, 6, 9],\n",
              "         [2, 4, 5]]])"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "TENSOR.ndim"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "OwTwjbGDf0nk",
        "outputId": "217df203-b0da-4349-d22f-93705e892ba6"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "3"
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "TENSOR.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xjffe97If9Q5",
        "outputId": "1e49105e-8024-41dd-e9ec-3a0680f00777"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([1, 3, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "TENSOR[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6NhtzkNygCGl",
        "outputId": "c2e46751-0a3c-4925-9156-dbfe23290286"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 2, 3],\n",
              "        [3, 6, 9],\n",
              "        [2, 4, 5]])"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Random tensors\n",
        "\n",
        "Why random tensors?\n",
        "\n",
        "Random tensors are important because the way many neural networks learn is that they start with tensors full of random numbers and then adjust those random numbers to better represent the data.\n",
        "\n",
        "`Start with random numbers -> look at data -> update random numbers -> look at data -> update random numbers`\n",
        "\n",
        "Torch random tensors - https://pytorch.org/docs/stable/generated/torch.rand.html"
      ],
      "metadata": {
        "id": "Sngk_9IXhUrI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a random tensor of size (3, 4)\n",
        "random_tensor = torch.rand(3, 4)\n",
        "random_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XC_LUjQQhsZN",
        "outputId": "68bfbdbb-7e7a-461d-fbab-0054b16cd3ac"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.4433, 0.7119, 0.4170, 0.4409],\n",
              "        [0.8014, 0.2050, 0.3547, 0.6358],\n",
              "        [0.3007, 0.1659, 0.3462, 0.7317]])"
            ]
          },
          "metadata": {},
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a random tensor with similar shape to an image tensor\n",
        "random_image_size_tensor = torch.rand(size=(3, 224, 224)) # height, width, colour channels (R, G, B)\n",
        "random_image_size_tensor.shape, random_image_size_tensor.ndim"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HNb5EHcyh_9v",
        "outputId": "229c6c05-867b-4bbb-9c8e-5c03554c3850"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(torch.Size([3, 224, 224]), 3)"
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Zeros and ones"
      ],
      "metadata": {
        "id": "U2FL8Tqkh_6s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor of all zeros\n",
        "zeros = torch.zeros(size=(3, 4))\n",
        "zeros"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PXLbKXX7h_4X",
        "outputId": "41735321-efc6-465f-ff00-7907f19c2640"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0.],\n",
              "        [0., 0., 0., 0.]])"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor of all ones\n",
        "ones = torch.ones(size=(3, 4))\n",
        "ones"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zvsnxHclh_1T",
        "outputId": "735cb920-e2bb-423d-bb8a-7a2812623d68"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 1., 1., 1.],\n",
              "        [1., 1., 1., 1.],\n",
              "        [1., 1., 1., 1.]])"
            ]
          },
          "metadata": {},
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "ones.dtype"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HgZwd5SLh_yQ",
        "outputId": "54785d1f-2d65-4108-99dc-dc7c87497c4e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.float32"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "random_tensor.dtype"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O58wXELFh_vr",
        "outputId": "eb3b1a05-e281-4857-8887-3ab4b049f80b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.float32"
            ]
          },
          "metadata": {},
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating a range of tensors and tensors-like "
      ],
      "metadata": {
        "id": "A9oyiP_yh_tG"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use torch.range() and get deprecated message, use torch.arange()\n",
        "one_to_ten = torch.arange(start=1, end=11, step=1)\n",
        "one_to_ten"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MXv_6ZRzkg48",
        "outputId": "c33aa934-d620-4be9-e4b4-b8414802e92b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10])"
            ]
          },
          "metadata": {},
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Creating tensors like\n",
        "ten_zeros = torch.zeros_like(input=one_to_ten)\n",
        "ten_zeros"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8jnWbY-ikhgd",
        "outputId": "d4bb2850-51b7-46a9-e4d0-39c5e5f98163"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Tensor datatypes \n",
        "\n",
        "**Note:** Tensor datatypes is one of the 3 big errors you'll run into with PyTorch & deep learning:\n",
        "1. Tensors not right datatype\n",
        "2. Tensors not right shape\n",
        "3. Tensors not on the right device\n",
        "\n",
        "Precision in computing - https://en.wikipedia.org/wiki/Precision_(computer_science)#:~:text=In%20computer%20science%2C%20the%20precision,used%20to%20express%20a%20value."
      ],
      "metadata": {
        "id": "gZAIvQyzm2E2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Float 32 tensor\n",
        "float_32_tensor = torch.tensor([3.0, 6.0, 9.0],\n",
        "                               dtype=None, # what datatype is the tensor (e.g. float32 or float16)\n",
        "                               device=None, # What device is your tensor on\n",
        "                               requires_grad=False) # whether or not to track gradients with this tensors operations\n",
        "float_32_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TEyHrEWFnO78",
        "outputId": "1fcbfb44-82b3-4eb6-dd15-473d5235acd7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([3., 6., 9.])"
            ]
          },
          "metadata": {},
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "float_32_tensor.dtype"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wke5_RItnWxS",
        "outputId": "89660a90-feba-4423-d79d-7885bc66cfb4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.float32"
            ]
          },
          "metadata": {},
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "float_16_tensor = float_32_tensor.type(torch.float16)\n",
        "float_16_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vqKcjujanY2J",
        "outputId": "45f42da5-4f86-4850-8337-58aff436a9cb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([3., 6., 9.], dtype=torch.float16)"
            ]
          },
          "metadata": {},
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "float_16_tensor * float_32_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JBerGK0MpkFA",
        "outputId": "a856bc6a-2507-45da-9976-b87e86f54069"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 9., 36., 81.])"
            ]
          },
          "metadata": {},
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "int_32_tensor = torch.tensor([3, 6, 9], dtype=torch.long)\n",
        "int_32_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NpgQSORsp9J6",
        "outputId": "a3b257ce-6215-4f43-838a-55dfd1807f65"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([3, 6, 9])"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "float_32_tensor * int_32_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9A6eAgP1qQLO",
        "outputId": "bcb9497b-42d4-47cd-ffea-cd2e987bd09e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 9., 36., 81.])"
            ]
          },
          "metadata": {},
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Getting information from tensors (tensor attributes)\n",
        "\n",
        "1. Tensors not right datatype - to do get datatype from a tensor, can use `tensor.dtype`\n",
        "2. Tensors not right shape - to get shape from a tensor, can use `tensor.shape`\n",
        "3. Tensors not on the right device - to get device from a tensor, can use `tensor.device`"
      ],
      "metadata": {
        "id": "g-JhM8bzqUTP"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor\n",
        "some_tensor = torch.rand(3, 4)\n",
        "some_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MTmy1K8NrA7q",
        "outputId": "ebce8bc4-9e1d-4040-df96-6343ba3c9354"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.7151, 0.9288, 0.0464, 0.2910],\n",
              "        [0.7281, 0.5272, 0.9098, 0.3145],\n",
              "        [0.9641, 0.4652, 0.8553, 0.0232]])"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find out details about some tensor\n",
        "print(some_tensor)\n",
        "print(f\"Datatype of tensor: {some_tensor.dtype}\")\n",
        "print(f\"Shape of tensor: {some_tensor.shape}\")\n",
        "print(f\"Device tensor is on: {some_tensor.device}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EVdSeMN9rG3t",
        "outputId": "02cb4a54-1699-49b2-d5be-b095aaac92ef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0.7151, 0.9288, 0.0464, 0.2910],\n",
            "        [0.7281, 0.5272, 0.9098, 0.3145],\n",
            "        [0.9641, 0.4652, 0.8553, 0.0232]])\n",
            "Datatype of tensor: torch.float32\n",
            "Shape of tensor: torch.Size([3, 4])\n",
            "Device tensor is on: cpu\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Manipulating Tensors (tensor operations)\n",
        "\n",
        "Tensor opertions include:\n",
        "* Addition\n",
        "* Subtraction\n",
        "* Multiplication (element-wise)\n",
        "* Division\n",
        "* Matrix multiplication"
      ],
      "metadata": {
        "id": "vGncmTNOrsEN"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor and add 10 to it\n",
        "tensor = torch.tensor([1, 2, 3])\n",
        "tensor + 10"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1UEudneS6Opn",
        "outputId": "92387bfa-caed-4525-8c64-7b07fdeee104"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([11, 12, 13])"
            ]
          },
          "metadata": {},
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Multiply tensor by 10\n",
        "tensor * 10"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7_a_8nnJ6Xsy",
        "outputId": "170fc190-01f9-41a3-c8b3-6509e2c23cca"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([10, 20, 30])"
            ]
          },
          "metadata": {},
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "363dnmSi6fg5",
        "outputId": "c7dc4a2d-210f-431a-d2fd-ce779312ba64"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1, 2, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 35
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Substract 10\n",
        "tensor - 10"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ab5Ijquc6gn5",
        "outputId": "3855276a-a447-40c8-ddeb-8b836b531e76"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([-9, -8, -7])"
            ]
          },
          "metadata": {},
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Try out PyTorch in-built functions\n",
        "torch.mul(tensor, 10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KlwwWMwQ6s7Z",
        "outputId": "805f1fdc-1208-4a3f-8615-9c87eeeea8ca"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([10, 20, 30])"
            ]
          },
          "metadata": {},
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "torch.add(tensor, 10)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KxV6flg_604F",
        "outputId": "8513f4c4-68ac-42cc-ecbd-0f51f40052f4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([11, 12, 13])"
            ]
          },
          "metadata": {},
          "execution_count": 38
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Matrix multiplication\n",
        "\n",
        "Two main ways of performing multiplication in neural networks and deep learning:\n",
        "\n",
        "1. Element-wise multiplication\n",
        "2. Matrix mutliplication (dot product)\n",
        "\n",
        "More information on multiplying matrices - https://www.mathsisfun.com/algebra/matrix-multiplying.html\n",
        "\n",
        "There are two main rules that performing matrix mutliplication needs to satisfy:\n",
        "1. The **inner dimensions** must match:\n",
        "* `(3, 2) @ (3, 2)` won't work\n",
        "* `(2, 3) @ (3, 2)` will work\n",
        "* `(3, 2) @ (2, 3)` will work\n",
        "2. The resulting matrix has the shape of the **outer dimensions**:\n",
        "* `(2, 3) @ (3, 2)` -> `(2, 2)`\n",
        "* `(3, 2) @ (2, 3)` -> `(3, 3)`"
      ],
      "metadata": {
        "id": "2HLSVYm6-bXw"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Element wise multiplication\n",
        "print(tensor, \"*\", tensor)\n",
        "print(f\"Equals: {tensor * tensor}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AL0Ef09oAPwz",
        "outputId": "8ffd729e-7b8f-4b25-934a-3485a88fef5f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([1, 2, 3]) * tensor([1, 2, 3])\n",
            "Equals: tensor([1, 4, 9])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Matrix multiplication\n",
        "torch.matmul(tensor, tensor)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "MVinO8q8AVgu",
        "outputId": "67961e52-a953-41e7-9bab-823521bf31a0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(14)"
            ]
          },
          "metadata": {},
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JQqozJdQA1dP",
        "outputId": "2174de30-0909-41bc-ee0d-f09ee534ea69"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1, 2, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 41
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Matrix multiplication by hand\n",
        "1*1 + 2*2 + 3*3"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "euQ9rTuvAl5y",
        "outputId": "a87fc8d2-96a5-4135-99c5-9f165ea3e971"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "14"
            ]
          },
          "metadata": {},
          "execution_count": 42
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "value = 0\n",
        "for i in range(len(tensor)):\n",
        "  value += tensor[i] * tensor[i]\n",
        "print(value)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ay6a2Ti1A9mc",
        "outputId": "cdf67c77-09f2-4269-afa4-68e617609a84"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(14)\n",
            "CPU times: user 661 µs, sys: 876 µs, total: 1.54 ms\n",
            "Wall time: 1.55 ms\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "%%time\n",
        "torch.matmul(tensor, tensor)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jBp_AC3YBTXX",
        "outputId": "bab5b7cb-acc2-49b1-b8fd-3a545e0148da"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "CPU times: user 67 µs, sys: 29 µs, total: 96 µs\n",
            "Wall time: 101 µs\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(14)"
            ]
          },
          "metadata": {},
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### One of the most common errors in deep learning: shape errors"
      ],
      "metadata": {
        "id": "VmX8MOCoBZMu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Shapes for matrix multiplication \n",
        "tensor_A = torch.tensor([[1, 2],\n",
        "                         [3, 4],\n",
        "                         [5, 6]])\n",
        "\n",
        "tensor_B = torch.tensor([[7, 10],\n",
        "                         [8, 11],\n",
        "                         [9, 12]])\n",
        "\n",
        "# torch.mm(tensor_A, tensor_B) # torch.mm is the same as torch.matmul (it's an alias for writing less code)\n",
        "torch.matmul(tensor_A, tensor_B)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 203
        },
        "id": "M2rpP-bxCmef",
        "outputId": "2ded8074-f6a0-465a-e6eb-720a5a70dec2"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-46-281a2d72c2ec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;31m# torch.mm(tensor_A, tensor_B) # torch.mm is the same as torch.matmul (it's an alias for writing less code)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatmul\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensor_A\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtensor_B\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m: mat1 and mat2 shapes cannot be multiplied (3x2 and 3x2)"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tensor_B.T"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zzrun--YHgzV",
        "outputId": "e73a749b-508b-4124-a3a2-2e41a840e4df"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[ 7,  8,  9],\n",
              "        [10, 11, 12]])"
            ]
          },
          "metadata": {},
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tensor_A.shape, tensor_B.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Iskru-L3FQyy",
        "outputId": "f2f2c310-b5ff-4835-8d09-93b5005d82e4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(torch.Size([3, 2]), torch.Size([3, 2]))"
            ]
          },
          "metadata": {},
          "execution_count": 48
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "To fix our tensor shape issues, we can manipulate the shape of one of our tensors using a **transpose**.\n",
        "\n",
        "A **transpose** switches the axes or dimensions of a given tensor."
      ],
      "metadata": {
        "id": "qd5aRZjgFUlV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "tensor_B, tensor_B.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gaPS2iZSFnkY",
        "outputId": "67f9f7fe-4877-4cf8-8bd5-8065bf686dc4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[ 7, 10],\n",
              "         [ 8, 11],\n",
              "         [ 9, 12]]), torch.Size([3, 2]))"
            ]
          },
          "metadata": {},
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tensor_B.T, tensor_B.T.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8HPpvbCSFkWs",
        "outputId": "e351a3d7-3e17-416f-ed93-80ffd30cdfa8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[ 7,  8,  9],\n",
              "         [10, 11, 12]]), torch.Size([2, 3]))"
            ]
          },
          "metadata": {},
          "execution_count": 50
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# The matrix multiplication operation works when tensor_B is transposed\n",
        "print(f\"Original shapes: tensor_A = {tensor_A.shape}, tensor_B = {tensor_B.shape}\")\n",
        "print(f\"New shapes: tensor_A = {tensor_A.shape} (same shape as above), tensor_B.T = {tensor_B.T.shape}\")\n",
        "print(f\"Multiplying: {tensor_A.shape} @ {tensor_B.T.shape} <- inner dimensions must match\")\n",
        "print(\"Output:\\n\")\n",
        "output = torch.matmul(tensor_A, tensor_B.T)\n",
        "print(output) \n",
        "print(f\"\\nOutput shape: {output.shape}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yIcUJL_DFqrs",
        "outputId": "1c7f4db8-5ac0-49c1-d983-e93fb1b5bf0c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Original shapes: tensor_A = torch.Size([3, 2]), tensor_B = torch.Size([3, 2])\n",
            "New shapes: tensor_A = torch.Size([3, 2]) (same shape as above), tensor_B.T = torch.Size([2, 3])\n",
            "Multiplying: torch.Size([3, 2]) @ torch.Size([2, 3]) <- inner dimensions must match\n",
            "Output:\n",
            "\n",
            "tensor([[ 27,  30,  33],\n",
            "        [ 61,  68,  75],\n",
            "        [ 95, 106, 117]])\n",
            "\n",
            "Output shape: torch.Size([3, 3])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Finding the min, max, mean, sum, etc (tensor aggregation)"
      ],
      "metadata": {
        "id": "-2Q9lQ10F-sx"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor\n",
        "x = torch.arange(1, 100, 10)\n",
        "x, x.dtype"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vmGm37iiIx_6",
        "outputId": "f662f4e9-5524-4871-8c2c-35aa30e61636"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([ 1, 11, 21, 31, 41, 51, 61, 71, 81, 91]), torch.int64)"
            ]
          },
          "metadata": {},
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the min\n",
        "torch.min(x), x.min()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "l7KbfOFeI8Wn",
        "outputId": "10715772-4bcd-40c4-ef9a-a004800f6bf8"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(1), tensor(1))"
            ]
          },
          "metadata": {},
          "execution_count": 53
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the max\n",
        "torch.max(x), x.max()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eQk93EJaI-j8",
        "outputId": "fff58ef6-be2e-40d0-8050-6de4393d3e18"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(91), tensor(91))"
            ]
          },
          "metadata": {},
          "execution_count": 54
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the mean - note: the torch.mean() function requires a tensor of float32 datatype to work\n",
        "torch.mean(x.type(torch.float32)), x.type(torch.float32).mean()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "YdNwHqMBJC-n",
        "outputId": "b2d38991-b734-468f-c79a-c01fc0b6254d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(46.), tensor(46.))"
            ]
          },
          "metadata": {},
          "execution_count": 55
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the sum\n",
        "torch.sum(x), x.sum()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0N9ppySsJHVY",
        "outputId": "6b8603f0-f16f-4b79-9379-64c1e20273ee"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(460), tensor(460))"
            ]
          },
          "metadata": {},
          "execution_count": 56
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Finding the positional min and max"
      ],
      "metadata": {
        "id": "kLa91TZiJ31t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "x"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eIG9GRbyKt0t",
        "outputId": "7be0fa2e-b125-4b11-a77b-7fd7aae2a4bf"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 1, 11, 21, 31, 41, 51, 61, 71, 81, 91])"
            ]
          },
          "metadata": {},
          "execution_count": 57
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the position in tensor that has the minimum value with argmin() -> returns index position of targt tensor where the minimum value occurs \n",
        "x.argmin()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LfYi96igKsQT",
        "outputId": "10ef9563-7789-4d8e-9bd1-a6091e5c27f4"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(0)"
            ]
          },
          "metadata": {},
          "execution_count": 58
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Jc5GKGO6KtWp",
        "outputId": "89a9dfaf-581d-484d-e42a-b6de95584256"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(1)"
            ]
          },
          "metadata": {},
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Find the position in tensor that has the maximum value with argmax()\n",
        "x.argmax()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "FBGnJblVKxu1",
        "outputId": "77334a4f-ffd7-4ba4-e700-fe21750102b0"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(9)"
            ]
          },
          "metadata": {},
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x[9]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "X9QQXAKaLEJ_",
        "outputId": "b30e9115-23d7-4429-ac9a-e6119060c966"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(91)"
            ]
          },
          "metadata": {},
          "execution_count": 61
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Reshaping, stacking, squeezing and unsqueezing tensors\n",
        "\n",
        "* Reshaping - reshapes an input tensor to a defined shape\n",
        "* View - Return a view of an input tensor of certain shape but keep the same memory as the original tensor\n",
        "* Stacking - combine multiple tensors on top of each other (vstack) or side by side (hstack)\n",
        "* Squeeze - removes all `1` dimensions from a tensor\n",
        "* Unsqueeze - add a `1` dimension to a target tensor\n",
        "* Permute - Return a view of the input with dimensions permuted (swapped) in a certain way"
      ],
      "metadata": {
        "id": "ZPcyfriKLHXT"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's create a tensor\n",
        "import torch\n",
        "x = torch.arange(1., 10.)\n",
        "x, x.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vRJedWy3LLjK",
        "outputId": "d616e6f4-2d72-41e9-b165-e277335b3cca"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([1., 2., 3., 4., 5., 6., 7., 8., 9.]), torch.Size([9]))"
            ]
          },
          "metadata": {},
          "execution_count": 77
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Add an extra dimension\n",
        "x_reshaped = x.reshape(1, 9)\n",
        "x_reshaped, x_reshaped.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9scaX6wcMcv_",
        "outputId": "4037fb10-0443-4752-9ca0-9f91a9d48946"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[1., 2., 3., 4., 5., 6., 7., 8., 9.]]), torch.Size([1, 9]))"
            ]
          },
          "metadata": {},
          "execution_count": 78
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Change the view \n",
        "z = x.view(1, 9)\n",
        "z, z.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-cQDwoKGMcqC",
        "outputId": "53f6d9f7-1489-4b11-c7a7-6dd9f28ce7d7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[1., 2., 3., 4., 5., 6., 7., 8., 9.]]), torch.Size([1, 9]))"
            ]
          },
          "metadata": {},
          "execution_count": 79
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Changing z changes x (because a view of a tensor shares the same memory as the original input)\n",
        "z[:, 0] = 5\n",
        "z, x"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x4ZmA_h4McRP",
        "outputId": "7fb6bdd8-077f-485a-ebde-91b2c36dd46f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.]]),\n",
              " tensor([5., 2., 3., 4., 5., 6., 7., 8., 9.]))"
            ]
          },
          "metadata": {},
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Stack tensors on top of each other\n",
        "x_stacked = torch.stack([x, x, x, x], dim=0)\n",
        "x_stacked"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "u1Fi833cNudG",
        "outputId": "f071a3d6-37bf-415c-b6ba-55c4655baede"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.],\n",
              "        [5., 2., 3., 4., 5., 6., 7., 8., 9.],\n",
              "        [5., 2., 3., 4., 5., 6., 7., 8., 9.],\n",
              "        [5., 2., 3., 4., 5., 6., 7., 8., 9.]])"
            ]
          },
          "metadata": {},
          "execution_count": 87
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# torch.squeeze() - removes all single dimensions from a target tensor\n",
        "print(f\"Previous tensor: {x_reshaped}\")\n",
        "print(f\"Previous shape: {x_reshaped.shape}\")\n",
        "\n",
        "# Remove extra dimensions from x_reshaped\n",
        "x_squeezed = x_reshaped.squeeze()\n",
        "print(f\"\\nNew tensor: {x_squeezed}\")\n",
        "print(f\"New shape: {x_squeezed.shape}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "F6XzyHuMOIJb",
        "outputId": "9ffba5d6-ee6a-4c09-a052-fb084a1afc67"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Previous tensor: tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.]])\n",
            "Previous shape: torch.Size([1, 9])\n",
            "\n",
            "New tensor: tensor([5., 2., 3., 4., 5., 6., 7., 8., 9.])\n",
            "New shape: torch.Size([9])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# torch.unsqueeze() - adds a single dimension to a target tensor at a specific dim (dimension)\n",
        "print(f\"Previous target: {x_squeezed}\")\n",
        "print(f\"Previous shape: {x_squeezed.shape}\")\n",
        "\n",
        "# Add an extra dimension with unsqueeze\n",
        "x_unsqueezed = x_squeezed.unsqueeze(dim=0)\n",
        "print(f\"\\nNew tensor: {x_unsqueezed}\")\n",
        "print(f\"New shape: {x_unsqueezed.shape}\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J9UTmuhvPGwV",
        "outputId": "bb3f01dc-33a3-44f5-d042-2adcc4bb202b"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Previous target: tensor([5., 2., 3., 4., 5., 6., 7., 8., 9.])\n",
            "Previous shape: torch.Size([9])\n",
            "\n",
            "New tensor: tensor([[5., 2., 3., 4., 5., 6., 7., 8., 9.]])\n",
            "New shape: torch.Size([1, 9])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# torch.permute - rearranges the dimensions of a target tensor in a specified order\n",
        "x_original = torch.rand(size=(224, 224, 3)) # [height, width, colour_channels]\n",
        "\n",
        "# Permute the original tensor to rearrange the axis (or dim) order\n",
        "x_permuted = x_original.permute(2, 0, 1) # shifts axis 0->1, 1->2, 2->0\n",
        "\n",
        "print(f\"Previous shape: {x_original.shape}\") \n",
        "print(f\"New shape: {x_permuted.shape}\") # [colour_channels, height, width]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rBGYuy2TPHra",
        "outputId": "bf9d9a76-0156-4a80-8538-f57a462a3b1d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Previous shape: torch.Size([224, 224, 3])\n",
            "New shape: torch.Size([3, 224, 224])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "x_original[0, 0, 0] = 728218\n",
        "x_original[0, 0, 0], x_permuted[0, 0, 0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "x5uR_Il-PQOF",
        "outputId": "e7868c1f-9475-4740-c594-698a4a43a8cb"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor(728218.), tensor(728218.))"
            ]
          },
          "metadata": {},
          "execution_count": 104
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Indexing (selecting data from tensors)\n",
        "\n",
        "Indexing with PyTorch is similar to indexing with NumPy."
      ],
      "metadata": {
        "id": "r8P5o0gdRh0x"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor\n",
        "import torch\n",
        "x = torch.arange(1, 10).reshape(1, 3, 3)\n",
        "x, x.shape"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "okvAfJ3ASinI",
        "outputId": "ec5155c1-cada-4e4f-d9f9-cbdb2ea361e3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([[[1, 2, 3],\n",
              "          [4, 5, 6],\n",
              "          [7, 8, 9]]]), torch.Size([1, 3, 3]))"
            ]
          },
          "metadata": {},
          "execution_count": 105
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's index on our new tensor\n",
        "x[0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "5hU35njXSnPi",
        "outputId": "10cdbc82-c7d1-49d3-f7cc-ab0fcec81ed7"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 2, 3],\n",
              "        [4, 5, 6],\n",
              "        [7, 8, 9]])"
            ]
          },
          "metadata": {},
          "execution_count": 106
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's index on the middle bracket (dim=1)\n",
        "x[0][0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pqFyG5S7SnKS",
        "outputId": "ead28195-f9b2-4944-9308-1154ad2a2e75"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1, 2, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 109
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's index on the most inner bracket (last dimension)\n",
        "x[0][1][1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Pf6GfxwBSnDL",
        "outputId": "bf6c471f-a5c9-4d32-83fb-6af71bd2bf5e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(5)"
            ]
          },
          "metadata": {},
          "execution_count": 113
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# You can also use \":\" to select \"all\" of a target dimension\n",
        "x[:, 0]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PQH4EttRSmyv",
        "outputId": "32d699f8-36ba-4f3f-9cb3-758f27d1cb15"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 2, 3]])"
            ]
          },
          "metadata": {},
          "execution_count": 114
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get all values of 0th and 1st dimensions but only index 1 of 2nd dimension\n",
        "x[:, :, 1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cdZcxA7ITuyt",
        "outputId": "94ba1d6b-2a40-44e5-bff1-fb75c6fe8b83"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[2, 5, 8]])"
            ]
          },
          "metadata": {},
          "execution_count": 115
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get all values of the 0 dimension but only the 1 index value of 1st and 2nd dimension\n",
        "x[:, 1, 1]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "hqv2hRz7T245",
        "outputId": "1894d5f5-1c9b-4b75-8c17-97e77e709557"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([5])"
            ]
          },
          "metadata": {},
          "execution_count": 118
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Get index 0 of 0th and 1st dimension and all values of 2nd dimension\n",
        "x[0, 0, :]"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QSXRblctUGTw",
        "outputId": "dc5ea85d-0d03-4774-b27f-a68ed219aa31"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1, 2, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 119
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Index on x to return 9\n",
        "print(x[0][2][2])\n",
        "\n",
        "# Index on x to return 3, 6, 9\n",
        "print(x[:, :, 2])"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DOCotUAhUT0R",
        "outputId": "caaeb65c-af20-4d2e-ecfc-1ec2923398ec"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor(9)\n",
            "tensor([[3, 6, 9]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## PyTorch tensors & NumPy\n",
        "\n",
        "NumPy is a popular scientific Python numerical computing library. \n",
        "\n",
        "And because of this, PyTorch has functionality to interact with it.\n",
        "\n",
        "* Data in NumPy, want in PyTorch tensor -> `torch.from_numpy(ndarray)`\n",
        "* PyTorch tensor -> NumPy -> `torch.Tensor.numpy()` "
      ],
      "metadata": {
        "id": "1OjawP9XUcEr"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# NumPy array to tensor\n",
        "import torch\n",
        "import numpy as np\n",
        "\n",
        "array = np.arange(1.0, 8.0)\n",
        "tensor = torch.from_numpy(array) # warning: when converting from numpy -> pytorch, pytorch reflects numpy's default datatype of float64 unless specified otherwise\n",
        "array, tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2I1qZb9dU150",
        "outputId": "306cd520-ffe8-4915-b315-9e49ca67994f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([1., 2., 3., 4., 5., 6., 7.]),\n",
              " tensor([1., 2., 3., 4., 5., 6., 7.], dtype=torch.float64))"
            ]
          },
          "metadata": {},
          "execution_count": 135
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Change the value of array, what will this do to `tensor`?\n",
        "array = array + 1\n",
        "array, tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D2D2_7EgU104",
        "outputId": "98fbe0f6-87ca-49c4-98c5-4ecb3a88fef1"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([2., 3., 4., 5., 6., 7., 8.]),\n",
              " tensor([1., 2., 3., 4., 5., 6., 7.], dtype=torch.float64))"
            ]
          },
          "metadata": {},
          "execution_count": 136
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Tensor to NumPy array\n",
        "tensor = torch.ones(7)\n",
        "numpy_tensor = tensor.numpy()\n",
        "tensor, numpy_tensor"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uVX9OHluU1v9",
        "outputId": "b655993d-f0dd-4361-efb8-644a0f709262"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([1., 1., 1., 1., 1., 1., 1.]),\n",
              " array([1., 1., 1., 1., 1., 1., 1.], dtype=float32))"
            ]
          },
          "metadata": {},
          "execution_count": 137
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Change the tesnor, what happens to `numpy_tensor`?\n",
        "tensor = tensor + 1\n",
        "tensor, numpy_tensor "
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "olqJ6oFtU1hF",
        "outputId": "17235a8b-fb54-4b2c-dc1d-55f4b6f77e46"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(tensor([2., 2., 2., 2., 2., 2., 2.]),\n",
              " array([1., 1., 1., 1., 1., 1., 1.], dtype=float32))"
            ]
          },
          "metadata": {},
          "execution_count": 139
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Reproducbility (trying to take random out of random)\n",
        "\n",
        "In short how a neural network learns:\n",
        "\n",
        "`start with random numbers -> tensor operations -> update random numbers to try and make them better representations of the data -> again -> again -> again...`\n",
        "\n",
        "To reduce the randomness in neural networks and PyTorch comes the concept of a **random seed**.\n",
        "\n",
        "Essentially what the random seed does is \"flavour\" the randomness."
      ],
      "metadata": {
        "id": "cWYQ6KRjW7cu"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "\n",
        "# Create two random tensors\n",
        "random_tensor_A = torch.rand(3, 4)\n",
        "random_tensor_B = torch.rand(3, 4)\n",
        "\n",
        "print(random_tensor_A)\n",
        "print(random_tensor_B)\n",
        "print(random_tensor_A == random_tensor_B)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "inL6FZxgYBYa",
        "outputId": "c3fdab3b-fb6d-4476-d019-41a3246b965e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0.3675, 0.8410, 0.0507, 0.3165],\n",
            "        [0.7275, 0.9676, 0.3901, 0.8840],\n",
            "        [0.5177, 0.2239, 0.4362, 0.3602]])\n",
            "tensor([[0.5229, 0.6719, 0.2790, 0.8198],\n",
            "        [0.6689, 0.8659, 0.7849, 0.4268],\n",
            "        [0.2076, 0.8076, 0.4377, 0.2555]])\n",
            "tensor([[False, False, False, False],\n",
            "        [False, False, False, False],\n",
            "        [False, False, False, False]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Let's make some random but reproducible tensors\n",
        "import torch\n",
        "\n",
        "# Set the random seed\n",
        "RANDOM_SEED = 42\n",
        "torch.manual_seed(RANDOM_SEED)\n",
        "random_tensor_C = torch.rand(3, 4)\n",
        "\n",
        "torch.manual_seed(RANDOM_SEED)\n",
        "random_tensor_D = torch.rand(3, 4)\n",
        "\n",
        "print(random_tensor_C)\n",
        "print(random_tensor_D)\n",
        "print(random_tensor_C == random_tensor_D)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EFGlNq_0Yskj",
        "outputId": "1697a05c-0a1d-4f9f-c704-52504955d173"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
            "        [0.3904, 0.6009, 0.2566, 0.7936],\n",
            "        [0.9408, 0.1332, 0.9346, 0.5936]])\n",
            "tensor([[0.8823, 0.9150, 0.3829, 0.9593],\n",
            "        [0.3904, 0.6009, 0.2566, 0.7936],\n",
            "        [0.9408, 0.1332, 0.9346, 0.5936]])\n",
            "tensor([[True, True, True, True],\n",
            "        [True, True, True, True],\n",
            "        [True, True, True, True]])\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Extra resources for reproducibility:\n",
        "* https://pytorch.org/docs/stable/notes/randomness.html\n",
        "* https://en.wikipedia.org/wiki/Random_seed"
      ],
      "metadata": {
        "id": "T8IDM0VTZdqA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        " ## Running tensors and PyTorch objects on the GPUs (and making faster computations)\n",
        "\n",
        " GPUs = faster computation on numbers, thanks to CUDA + NVIDIA hardware + PyTorch working behind the scenes to make everything hunky dory (good)."
      ],
      "metadata": {
        "id": "mw3pNVM9aSTb"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 1. Getting a GPU\n",
        "\n",
        "1. Easiest - Use Google Colab for a free GPU (options to upgrade as well)\n",
        "2. Use your own GPU - takes a little bit of setup and requires the investment of purchasing a GPU, there's lots of options..., see this post for what option to get: https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/\n",
        "3. Use cloud computing - GCP, AWS, Azure, these services allow you to rent computers on the cloud and access them\n",
        "\n",
        "For 2, 3 PyTorch + GPU drivers (CUDA) takes a little bit of setting up, to do this, refer to PyTorch setup documentation: https://pytorch.org/get-started/locally/ "
      ],
      "metadata": {
        "id": "Upp8e_kEfvhh"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!nvidia-smi"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iKzmq38ggcjS",
        "outputId": "c96c8540-593d-461c-e99b-57b98e6a549f"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Sun Feb 20 00:24:35 2022       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|                               |                      |               MIG M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   32C    P0    28W / 250W |      0MiB / 16280MiB |      0%      Default |\n",
            "|                               |                      |                  N/A |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                                  |\n",
            "|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |\n",
            "|        ID   ID                                                   Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 2. Check for GPU access with PyTorch"
      ],
      "metadata": {
        "id": "FrZo_oN_hBK4"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Check for GPU access with PyTorch\n",
        "import torch\n",
        "torch.cuda.is_available()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w1d1XxxvhT8h",
        "outputId": "df38af09-ffbf-4fe0-ae75-a14d8bab9da3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "For PyTorch since it's capable of running compute on the GPU or CPU, it's best practice to setup device agnostic code: https://pytorch.org/docs/stable/notes/cuda.html#best-practices\n",
        "\n",
        "E.g. run on GPU if available, else default to CPU"
      ],
      "metadata": {
        "id": "Q1apPaLTiOol"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Setup device agnostic code \n",
        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
        "device"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        },
        "id": "FxxX6NWLhhxb",
        "outputId": "875ab14a-5148-41d1-9691-b7401100d29d"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            },
            "text/plain": [
              "'cuda'"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Count number of devices\n",
        "torch.cuda.device_count()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nBTXPocmh1pD",
        "outputId": "c973fe94-3b34-4c23-feaf-0c19a139073e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {},
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## 3. Putting tensors (and models) on the GPU\n",
        "\n",
        "The reason we want our tensors/models on the GPU is because using a GPU results in faster computations."
      ],
      "metadata": {
        "id": "_CDYD4VHh9aL"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a tensor (default on the CPU)\n",
        "tensor = torch.tensor([1, 2, 3])\n",
        "\n",
        "# Tensor not on GPU\n",
        "print(tensor, tensor.device)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "GlbeX_rkiyP5",
        "outputId": "a9fbdbfe-c607-402f-a861-1ee6e4a1ce0e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "tensor([1, 2, 3]) cpu\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Move tensor to GPU (if available)\n",
        "tensor_on_gpu = tensor.to(device)\n",
        "tensor_on_gpu"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1AJny5yijCeG",
        "outputId": "12bbe2b0-26ca-4a04-b89e-79934a5671bd"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1, 2, 3], device='cuda:0')"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "### 4. Moving tensors back to the CPU"
      ],
      "metadata": {
        "id": "0Je1pAbGjPV1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# If tensor is on GPU, can't transform it to NumPy\n",
        "tensor_on_gpu.numpy()"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 185
        },
        "id": "6cappzQ9jl9v",
        "outputId": "58c42ada-af5a-4a2c-fdd7-6850dd672222"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "TypeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-8-b7da913938a5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# If tensor is on GPU, can't transform it to NumPy\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mtensor_on_gpu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mTypeError\u001b[0m: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first."
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# To fix the GPU tensor with NumPy issue, we can first set it to the CPU\n",
        "tensor_back_on_cpu = tensor_on_gpu.cpu().numpy()\n",
        "tensor_back_on_cpu"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rL0RrzfXjs7i",
        "outputId": "c0a7ca85-4909-4f39-8c9c-5de636c90ace"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 2, 3])"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "tensor_on_gpu"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DxRSKgKIj-FH",
        "outputId": "9a083db8-06cb-4051-f9b4-1dbe04c69637"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1, 2, 3], device='cuda:0')"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Exercises & Extra-curriculum\n",
        "\n",
        "See exercises for this notebook here: https://www.learnpytorch.io/00_pytorch_fundamentals/#exercises \n",
        "See the template exercises notebook for this module here: https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/exercises/00_pytorch_fundamentals_exercises.ipynb "
      ],
      "metadata": {
        "id": "49zOv23ZkC-s"
      }
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "1ZC_gjuYkz_B"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}